Unifying Divergence Minimization and Statistical Inference Via Convex Duality
نویسندگان
چکیده
In this paper we unify divergence minimization and statistical inference by means of convex duality. In the process of doing so, we prove that the dual of approximate maximum entropy estimation is maximum a posteriori estimation. Moreover, our treatment leads to stability and convergence bounds for many statistical learning problems. Finally, we show how an algorithm by Zhang can be used to solve this class of optimization problems efficiently.
منابع مشابه
t-divergence Based Approximate Inference
Approximate inference is an important technique for dealing with large, intractable graphical models based on the exponential family of distributions. We extend the idea of approximate inference to the t-exponential family by defining a new t-divergence. This divergence measure is obtained via convex duality between the log-partition function of the t-exponential family and a new t-entropy. We ...
متن کاملOn the duality of quadratic minimization problems using pseudo inverses
In this paper we consider the minimization of a positive semidefinite quadratic form, having a singular corresponding matrix $H$. We state the dual formulation of the original problem and treat both problems only using the vectors $x in mathcal{N}(H)^perp$ instead of the classical approach of convex optimization techniques such as the null space method. Given this approach and based on t...
متن کاملInformation Measures via Copula Functions
In applications of differential geometry to problems of parametric inference, the notion of divergence is often used to measure the separation between two parametric densities. Among them, in this paper, we will verify measures such as Kullback-Leibler information, J-divergence, Hellinger distance, -Divergence, … and so on. Properties and results related to distance between probability d...
متن کاملVariational Principle
Variational principle for probabilistic learning Yet another justification More simplification of updates for mean-field family Examples Dirichlet Process Mixture On minimization of divergence measures Energy minimization justifications Variational learning with exponential family Mean parametrization and marginal polytopes Convex dualities The log-partition function and conjugate duality Belie...
متن کاملMinimization Problems Based on a Parametric Family of Relative Entropies I: Forward Projection
Minimization problems with respect to a one-parameter family of generalized relative entropies are studied. These relative entropies, which we term relative α-entropies (denoted Iα), arise as redundancies under mismatched compression when cumulants of compressed lengths are considered instead of expected compressed lengths. These parametric relative entropies are a generalization of the usual r...
متن کامل